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EXECUTIVE  SUMMARY 


Most  “real-world”  problem  solving  is  multivariate,  complex,  and  steeped  in  uncer¬ 
tainty.  It  encompasses  a  full  range  of  activity— from  vehicle  maintenance  to  tactical 
deployments.  It  is  an  integral  and  inevitable  component  of  the  human  competence  needed  to 
ensure  workforce  readiness  and  viability  in  the  global  marketplace.  It  is  an  equally  integral 
and  inevitable  component  of  military  readiness. 

Given  the  complexity  of  real-world  problem  solving  and  the  range  (both  descriptive 
and  prescriptive)  of  its  theoretic  underpinnings  and  procedures,  it  seems  reasonable  to  seek 
assistance  from  technology.  The  value  of  doing  so  is  presaged  by  early  studies  of  clinical 
and  statistical  decision-making.  These  studies  found  statistical,  algorithmic  procedures  to 
be  superior  to  the  clinical  judgments  of  human  beings,  even  though  the  statistical  proce¬ 
dures  were  derived  solely  from  the  practice  of  human  decision-makers. 

This  study  reviewed  the  effectiveness  and  cost  benefits  of  technology  used  to  aid 
performance  of  maintenance  operations.  Performance  data  have  been  reported  for  three 
military  performance- aiding  systems: 

1.  Computer-Based  Maintenance  Aids  System  (CMAS).  CMAS  was 
developed  by  the  Air  Force  Armstrong  Laboratory  Human  Resources  Direc¬ 
torate,  Brooks  Air  Force  Base  (AFB),  Texas. 

2.  Portable  Electronic  Aid  for  Maintenance  (PE AM).  PEAM  was  devel¬ 
oped  by  the  U.S  Army  Research  Institute  for  Behavioral  and  Social  Sciences 
(ARI)  and  the  Navy  Personnel  Research  and  Development  Center  (NPRDC). 

3.  Integrated  Maintenance  Information  System  (IMIS).  IMIS  was 

developed  by  the  Air  Force  Armstrong  Laboratory  Human  Resources  Director¬ 
ate,  Brooks  AFB,  Texas. 

CMAS  was  a  logical  extension  of  efforts  to  develop  maintenance  performance  aids 
that  could  be  traced  at  least  to  the  1960s,  beginning  with  paper-based  performance  aids  and 
later  with  those  same  performance  aids  installed  on  computer  systems.  Evaluation  studies 
compared  technicians  using  paper-based  technical  manuals  (TMs)  with  those  using  CMAS. 
These  evaluations  found  that  technicians  using  CMAS  took  less  than  half  the  time  to  find 
system  faults,  were  able  to  check  more  test  points,  made  fewer  (i.e.,  no)  false  replace¬ 
ments,  and  solved  more  problems. 


ES-1 


PEAM  followed  the  Defense  Advanced  Research  Projects  Agency’s  (DARPA) 
development  of  the  Voice  Interactive  Maintenance  Aiding  Device  (VIMAD)  in  the  late 
1970s.  VIMAD  was  the  first  voice-controlled,  wearable  computer  intended  as  a  mainte¬ 
nance  performance  aid.  PEAM  was  also  portable  (briefcase-size)  and  used  voice  interaction 
to  allow  hands-free  access  to  text  and  graphics  maintenance  information  needed  by  techni¬ 
cians.  Evaluations,  which  for  a  variety  of  reasons  did  not  use  voice  interaction,  found  that 
maintenance  technicians  using  PEAM  exhibited  substantial  reductions  (factors  of  5  and  6) 
of  errors  in  troubleshooting  tasks  and  reductions  (factor  of  2)  in  errors  solving  nontrouble¬ 
shooting  problems. 

IMIS  is  a  more  recently  evaluated  computer-based  performance- aiding  system.  Its 
evaluation  concerned  fault-isolation  problems  in  three  F-16  avionics  subsystems:  the  fire 
control  radar  (FCR),  the  heads-up  display  (HUD),  and  the  Inertial  Navigation  System 
(INS).  Technicians  involved  in  the  evaluation  study  used  paper-based  task  orders  (TOs) 
(Air  Force  TMs)  for  half  of  the  problems  and  IMIS  for  the  other  half.  These  evaluations 
found  that  technicians  using  IMIS  found  more  correct  solutions  in  less  time,  used  fewer 
parts  to  do  so,  and  took  less  time  to  order  parts.  Analysis  of  costs  found  a  net  savings  of 
about  $20  million  per  year  in  maintaining  these  three  avionics  subsystems  for  the  full  Air 
Force  fleet  of  F-16s. 

These  findings  suggest  that 

•  A  strong  cost-effectiveness  case  can  be  made  for  using  these  computer-based 
performance- aiding  devices. 

•  The  development  and  implementation  of  these  devices  should  consider  the  full 
range  of  options  available  for  ensuring  competent  human  performance. 

•  Both  descriptive  and  prescriptive  approaches  should  be  employed  in  the  design 
of  these  devices. 

•  These  devices  should  capitalize  on  modeling  capabilities  developed  for  intelli¬ 
gent  tutoring  systems 

•  The  absence  of  these  devices  from  routine  use,  despite  their  demonstrated 
value,  suggests  that  more  effort  is  needed  to  ensure  that  the  state  of  practice  in 
maintenance  operations  advances  along  with  the  state  of  the  art. 
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EFFECTIVENESS  AND  COST  BENEFITS  OF 
COMPUTER-BASED  PERFORMANCE  AIDS 
FOR  EQUIPMENT  MAINTENANCE 


A.  INTRODUCTION 

Decision-making  and  problem  solving  by  technicians  are  central  components  of 
maintenance  operations.  Problem  solving  is  required  when  an  individual  or  a  group  of 
individuals  must  achieve  a  goal  but  are  uncertain  how  to  do  so  (Baker  and  Mayer,  1999; 
Mayer  and  Wittrock,  1996).  Problem  solvers  require  ingenuity  and  creativity  to  manipulate 
and  transform  the  knowledge  and  skills  they  possess  into  paths  of  action  leading  to  the 
goal.  Most  “real-world”  problem  solving  is  a  multivariate  and  complex  activity  steeped  in 
uncertainty.  It  involves  a  full  spectrum  of  activity— ranging  from  vehicle  maintenance  to 
tactical  deployments. 

Decision-making  is  an  integral  aspect  of  human  problem  solving.  It  is  a  critical  skill 
needed  to  ensure  workforce  readiness  and  viability  in  the  global  marketplace  (O’Neil, 
1999).  It  is  made  difficult  by  the  frequency  with  which  decision  makers  are  confronted 
with  too  much  data,  too  many  options,  and  unknown  levels  of  risk.  Decision-making  has 
been  the  object  of  systematic  study  by  past  and  present  psychologists  for  many  years 
(James,  1890/1950;  Edwards  and  Fasolo,  2001). 

Given  the  complexity  of  real-world  decision-making  and  problem  solving  and  the 
descriptive  and  prescriptive  range  of  their  theoretic  underpinnings,  seeking  assistance  from 
technology  in  both  activities  seems  reasonable.  The  value  of  technology  applications  in  this 
area  is  presaged  by  early  studies  of  clinical  and  statistical  decision-making  (Meehl,  1954). 
These  studies  were  originally  intended  to  show  what  human  (clinical)  judgment  would  add 
to  purely  statistical  predictions  of  outcomes,  such  as  patient  response  to  treatment  or  aca¬ 
demic  success.  As  described  by  Dawes  (1971),  the  statistical  prediction  “floor”  turned  out 
to  be  a  ceiling.  In  all  20  cases  reviewed  by  Meehl,  statistical  predictions  based  on  straight¬ 
forward  linear  models  turned  out  to  be  superior  to  the  clinical  judgments  of  human  beings. 

Meehl’s  results  might  be  taken  to  suggest  that  we  should  seek  to  replace  human 
decision-making  with  computer-based  algorithmic  procedures  that  capture  human  processes 
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but  apply  them  consistently.  However,  this  approach  may  go  too  far  for  at  least  three  rea¬ 
sons: 

1.  Many  decisions  must  be  made  in  a  dynamic  environment.  By  the  time  an  algo¬ 
rithmic  procedure  can  capture  the  environment  and  the  decision  process,  the 
need  for  the  decision  may  have  long  passed. 

2.  As  described  by  Hastie  (2001),  incorporating  all  the  elements  that  should  be 
included  in  a  decision  may  be  impossible.  Elements  involving  intuition,  social 
roles,  identification  of  alternatives,  payoff-probability  interactions,  utilities, 
uncertainties,  and  so  forth  are  often  too  elusive  to  be  captured  in  anything  like 
the  algorithms  needed  for  technological  applications. 

3.  Most  humans  want  to  maintain  control  over  their  lives.  They  do  not  want  their 
lives  to  be  run  by  machines. 

On  the  other  hand,  most  people  are  willing  to  accept  (even  pay  for)  assistance  and 
advice  in  making  complicated  decisions.  The  extent  to  which  they  are  willing  to  do  this 
depends  on  the  decision  to  be  made  and  the  individual  who  must  make  it.  Areas  in  which  a 
great  many  possibilities  can  be  collected,  stored,  and  accessed  by  computer  and  then  orga¬ 
nized  and  presented  to  human  decision-makers  seem  ripe  for  technology  assistance.  The 
value  of  using  computer-based  technology  to  aid  human  decision-making,  problem 
solving,  and  generally  to  augment  human  cognition  is  a  topic  of  this  document.  An  example 
of  such  augmentation  used  in  life  or  death  decisions  is  the  clinical  oncology  decision  aid. 

Example:  Clinical  Oncology  Decision  Aid 

A  combined  theoretical  approach  was  used  to  develop  the  National  Cancer  Insti¬ 
tute’s  (NCI)  Clinical  Trial  Decision  Aid  (Whiteis,  McGovern,  and  Johnston,  2001).  The 
decision  aid  uses  standard  personal  digital  assistant  (PDA)  technology  and  evolving  heuris¬ 
tics  for  inclusion  and  exclusion  to  assist  oncologists  in  determining  a  patient’s  eligibility  to 
participate  in  melanoma  and  colorectal  cancer  clinical  trials. 

Before  developing  this  decision  aid,  Mozelak,  Glassman,  and  Johnston  (2001)  per¬ 
formed  a  clinical  oncology  needs  assessment  that  included  interviews,  focus  groups,  and 
surveys.  They  found  that  14  percent  of  all  cancer  patients  are  eligible  for  enrollment  in  can¬ 
cer  clinical  trials  but  only  2  percent  are  actually  enrolled.  They  also  found  hundreds  of  pub¬ 
lic  and  private  clinical  trials  are  being  performed  nationally.  New  clinical  trials  are  added 
and  existing  trials  are  dropped  with  equal  frequency.  Each  clinical  trial  has  its  own  criteria 
for  including  and  excluding  patients,  and  each  is  subject  to  continuing  modification  by 
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regulatory  bodies  and  current  research  results.  The  situation  makes  enrollment  heuristics 
hard  to  acquire  and  apply. 

Physicians  indicated  that  using  the  Internet  or  journals  to  track  clinical  trials  in  their 
field  was  an  unmanageable  solution  that  proved  to  be  time  consuming  and  confusing 
because  of  rapid  changes  in  trial  availability.  They  also  indicated  that  each  trial  had  its  own 
selection  criteria,  which  left  them  to  focus  on  one  or  two  individual  trials,  neglect  other  tri¬ 
als,  and  seek  patients  who  fit  the  criteria  for  the  specific  trials  they  selected.  Mozelak, 
Glassman,  and  Johnston  (2001)  concluded  that  significant  problems  exist  in  enrolling  can¬ 
cer  patients  for  clinical  trials  because  of  the  need  to  manage  increasing  volumes  of  data  and 
track  the  rapidly  evolving  inclusion  and  exclusion  heuristics  for  each  trial. 

By  searching  clinical  trial  inclusion  and  exclusion  criteria,  Mozelak,  Glassman,  and 
Johnston  (2001)  determined  that  algorithmic  heuristics  could  fit  each  of  the  clinical  trials. 
Physicians  could  adjust  the  values  of  seven  clinical  variables,  thereby  reducing  the  number 
of  available  clinical  trial  choices  from  more  than  400  matched  trials  per  patient  to  an  average 
of  3  matched  trials  per  patient.  While  the  selection  rules  were  not  a  perfect  fit,  these  rules 
reduced  to  a  manageable  number  the  trials  for  which  any  patient  might  hypothetically  qual¬ 
ify.  In  turn,  the  process  increased  the  probability  that  individual  patients  might  qualify  for  a 
clinical  trial  because  the  physician  no  longer  needed  to  screen  them  for  a  limited  number  of 
trials  but  could  identify  for  each  patient  all  the  trials  that  were  appropriate  and  applicable. 

A  prototype  decision  aid  was  developed  on  a  handheld  PalmOS™  device  that 
downloads  NCI  melanoma  and  colorectal  cancer  clinical  trial  information  each  time  a  physi¬ 
cian  connects  the  handheld  device  to  the  Internet.  By  combining  formally  developed  heu¬ 
ristics  with  a  large  source  of  data,  the  device  can  locate  hundreds  of  clinical  trials  along 
with  their  descriptions,  locations,  points  of  contact,  and  inclusion  and  exclusion  criteria. 
The  device  is  currently  being  tested  against  other  clinical  datasets  to  verify  its  inclusion  and 
exclusion  heuristics. 

Recently,  80  percent  of  one  sample  of  oncologists  reported  using  PDA-based  deci¬ 
sion  aids  for  this  purpose  [Mozelak,  Glassman,  and  Johnston  (2001)].  Many  other  applica¬ 
tions  are  likely  to  be  developed,  raising  issues  concerning  their  cost  and  effectiveness. 
Their  value  is  indicated  by  the  military’s  assessments  of  problem-solving  and  decision- 
aiding  devices. 


3 


B .  MILITARY  APPLICATIONS  OF  TECHNOLOGY-BASED 
PERFORMANCE  AIDS 

University  research  is  primarily  concerned  with  developing  technical  opportunities, 
not  assessing  cost  and  effectiveness  tradeoffs.  This  orientation  leads  to  an  interest  in  effec¬ 
tiveness  (i.e.,  Does  it  work?)  but  not  necessarily  to  cost-effectiveness  (i.e.,  Should  anyone 
buy  it?).  Consideration  of  the  effectiveness  and  the  costs  of  a  proposed  innovation  is 
essential  for  its  transition  from  the  research  laboratory  to  routine  use.  The  primary  concern 
of  business  is  not  to  advance  the  state  of  the  art,  but  to  seek  proprietary  advantage.  This 
situation  leaves  the  task  of  enhancing  the  state  of  the  art  and  practice  in  performance  aiding 
to  the  government— particularly,  the  military.  The  task  requires  impartial  assessments  of 
costs  and  effectiveness  and  open  dissemination  of  the  findings. 

Three  system  developments  are  notable  in  this  regard: 

1.  Computer-Based  Maintenance  Aids  System  (CMAS).  CMAS  was 
developed  by  the  Air  Force  Armstrong  Laboratory  Human  Resources  Direc¬ 
torate,  Brooks  Air  Force  Base,  Texas. 

2.  Portable  Electronic  Aid  for  Maintenance  (PEAM).  PEAM  was  devel¬ 
oped  by  the  U.S  Army  Research  Institute  for  Behavioral  and  Social  Sciences 
(ARI)  and  the  Navy  Personnel  Research  and  Development  Center  (NPRDC). 

3.  Integrated  Maintenance  Information  System  (IMIS).  IMIS  was 

developed  by  the  Air  Force  Armstrong  Laboratory  Human  Resources  Director¬ 
ate,  Brooks  Air  Force  Base,  Texas. 

1.  CMAS 

CMAS  was  a  logical  extension  of  efforts  to  develop  maintenance  performance  aids 
that  could  be  traced  at  least  to  the  1960s,  beginning  with  an  Air  Force  Project  called  Pres¬ 
entation  of  Information  for  Maintenance  and  Operations  (PIMO)  (Serendipity,  1969). 
PIMO  used  paper-based,  task-specific  job  guides  as  performance  aids  for  maintenance  and 
was  followed  by  other  paper-based  performance  aids  such  as  XYZYX  Corporation’s  Job 
Performance  Aids  on  cards  (Inaba,  1988). 

Paper-based  performance  aids  were  shown  to  be  better  than  conventional  technical 
manuals  in  improving  technician  performance  (e.g.,  Foley  and  Camm,  1972;  Booher, 
1978),  but  they  shared  and  continue  to  share  the  usual  drawbacks  of  paper-based  technical 
manuals.  For  example,  they  are  expensive  and  inefficient  to  update;  it  is  difficult  to  design 
their  presentations  to  match  the  differing  needs  of  novice,  journeymen,  and  experienced 
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technicians;  they  often  make  necessary  information  difficult  to  find  and  access;  and  they  are 
heavy  and  cumbersome  to  store  and  use. 

Computer  technology  early  entered  the  scene  with  recipe  conversion  aids  presented 
by  the  Programmed  Logic  for  Automatic  Teaching  Operations  (PLATO)  instructional  sys¬ 
tem.  Hurlock  and  Slough  (1976)  reported  data  showing  that  the  capability  was  effective. 
However,  given  the  state  of  computer  technology  at  the  time,  PLATO  was  too  expensive 
and  too  cumbersome  for  routine  use.  In  1977,  the  Air  Force’s  CMAS  project  extended  this 
capability  (Clay,  1986).  CMAS  began  the  development  of  concepts  for  presenting  mainte¬ 
nance-aiding  information  by  computer  and  initiated  a  chain  of  developmental  efforts  con¬ 
tinuing  into  today’s  Interactive  Electronic  Technical  Manuals  (IETMs)  and  the  “mentoring” 
(decision-aiding)  capabilities  now  incorporated  in  the  Advanced  Distributed  Learning  Ini¬ 
tiative  (Dodds,  2000,  2001). 

Nugent  et  al.  (1987)  compared  the  troubleshooting  performance  of  36  technicians 
using  technical  manuals  (TMs)  with  technicians  using  electronic  presentation  of  an  aug¬ 
mented  CMAS  database  to  detect  and  isolate  single  component  failures  in  a  radio  receiver- 
transmitter.  Four  problems  were  presented:  two  had  to  be  solved  using  performance  aiding 
presented  by  TMs  and  two  had  to  be  solved  using  the  electronically  presented  performance 
aiding.  Table  1  shows  the  results  reported  by  Nugent  et  al.  As  the  table  suggests,  techni¬ 
cians  using  CMAS  took  less  than  half  the  time  to  find  system  faults,  were  able  to  check 
more  test  points,  made  no  false  replacements,  and  solved  more  problems.  All  these  results 
are  statistically  significant. 


Table  1.  Troubleshooting  Performance  of  36  Technicians  in  Locating  Two  Faults 
Using  Technical  Manuals  (TMs)  and  Two  Faults  Using  CMAS 


Avg.  Minutes  to 
Locate  Fault 

Avg.  Number  of 
Test  Points 

Checked 

Avg.  Number 
of  False 
Replacements 

Avg.  Number  of 
Problems  Solved 

TMs 

CMAS 

TMs 

CMAS 

TMs 

CMAS 

TMs 

CMAS 

56.5 

24.4 

3.6 

5.6 

1 .2 

0.0 

1.7 

2.0 

2.  PEAM 

PEAM  followed  the  Defense  Advanced  Research  Projects  Agency’s  (DARPA) 
development  of  the  Voice  Interactive  Maintenance  Aiding  Device  (VIMAD)  in  the  late 
1970s.  VIMAD  was  the  first  voice-controlled,  wearable  computer  intended  as  a  mainte¬ 
nance  performance  aid.  PEAM  also  was  portable  (briefcase-size)  and  used  voice  interaction 
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to  allow  hands-free  access  to  textual  and  graphics  maintenance  information  needed  by  tech¬ 
nicians.  For  a  variety  of  reasons,  these  evaluations  did  not  use  voice  interaction. 

Evaluation  of  PEAM  was  a  joint  Service  effort  summarized  by  Wisher  and  Kincaid 
(1989).  It  involved  Army  and  Navy  technicians.  The  Army  used  PEAM  and,  alternatively, 
a  laptop  computer  to  provide  PEAM-based  maintenance  performance  aiding  for  Ml  tank 
turrets.  The  Army  used  a  between-subjects  evaluation,  with  9  technicians  assigned  to  the 
PEAM  group  and  5  technicians  assigned  to  a  paper-based  TM  group.  Both  groups  of  tech¬ 
nicians  solved  6  troubleshooting  tasks  and  28  nontroubleshooting  tasks  (3  adjust  and  align 
tasks,  2  service  maintenance  tasks,  11  unit  removal  tasks,  and  12  install/replace  tasks). 
Table  2  presents  the  evaluation  results. 


Table  2.  Maintenance  Performance  of  Army  and  Navy  Technicians 
Using  Technical  Manuals  (TMs)  and  PEAM 


Avg.  Errors  per 
TS  Task 

Avg.  Errors  per 
NTS  Task 

Avg.  Minutes  per 
TS  Task 

Avg.  Minutes  per 
NTS  Task 

TMs 

PEAM 

TMs 

PEAM 

TMs 

PEAM 

TMs 

PEAM 

Army 

0.7 

1.1 

0.4 

37.0 

41 .6 

12.0 

16.1 

Navy 

5.7 

0.9 

N/A 

N/A 

43.9 

33.1 

N/A 

N/A 

Note  for  Table  2:  TS  =  troubleshooting;  NTS  =  nontroubleshooting. 


To  present  the  PEAM  material,  the  Navy  used  a  workstation- size  computer  to 
provide  PEAM-based  maintenance  performance  aiding  for  the  North  Atlantic  Treaty  Orga¬ 
nization  (NATO)  SEASPARROW  missile.  It  used  a  within-subjects  design  with  28  techni¬ 
cians  required  to  solve  two  fault-isolation  (troubleshooting  only)  problems:  one  set  of 
technicians  used  TMs  and  the  other  set  of  technicians  used  PEAM  simulation. 

Table  2  shows,  for  PEAM  groups,  substantial  reductions  (factors  of  5  and  6)  of 
errors  in  troubleshooting  tasks  for  the  Army  and  Navy  technicians  and  lesser,  but  still 
substantial,  reductions  (factor  of  2)  of  errors  among  the  Army  technicians  solving  nontrou¬ 
bleshooting  problems.  Ail  these  results  are  statistically  significant  (and  practically  signifi¬ 
cant  for  military  operations). 

Results  concerning  the  time  to  perform  tasks  were  mixed,  most  probably  [as 
Wisher  and  Kincaid  (1989)  suggest]  because  of  the  long  time  (in  excess  of  15  sec)  that  it 
took  for  graphics  to  appear  on  the  Army  PEAM  systems.  Consequently,  the  Army  techni¬ 
cians  using  PEAM  took  longer  to  perform  troubleshooting  and  nontroubleshooting  tasks, 
although  neither  of  these  differences  was  statistically  significant.  On  the  other  hand,  the 
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Navy  technicians,  who  used  a  more  powerful  computer  for  PEAM,  finished  their  trouble¬ 
shooting  tasks  more  quickly.  This  difference  is  statistically  significant. 

3.  IMIS 

Assessments  of  IMIS  provide  perhaps  the  best  and  most  complete  current  evidence 
on  the  value  of  technology -based  performance  aiding.  Tomasetti  et  al.  (1993)  documented  a 
thorough  cost  analysis  of  IMIS,  Thomas  (1995)  reported  results  from  an  empirical  investi¬ 
gation  of  IMIS  effectiveness,  and  Teitelbaum  and  Orlansky  (1996)  summarized  results 
from  both  studies,  combined  them  into  a  more  complete  cost-effectiveness  assessment,  and 
discussed  the  implications  of  these  findings. 

Thomas  (1995)  compared  the  performance  of  12  avionics  specialists  and  12  Air¬ 
plane  General  (APG)  technicians  on  12  fault-isolation  problems  concerning  three  F-16  avi¬ 
onics  subsystems:  the  fire  control  radar  (FCR),  the  heads-up  display  (HUD),  and  the 
Inertial  Navigation  System  (INS).  Within  each  of  the  two  groups  of  subjects,  six  of  the 
fault-isolation  problems  were  performed  using  paper-based  TOs  (Air  Force  TMs),  and  six 
were  performed  using  IMIS.  Training  for  APG  technicians  includes  all  aspects  of  aircraft 
maintenance,  only  a  small  portion  of  which  concerns  avionics.  In  contrast,  the  avionics 
specialists,  who  must  meet  higher  selection  standards,  receive  16  weeks  of  specialized 
training  focused  on  avionics  maintenance. 

Results  of  this  study,  which  are  shown  in  Table  3,  can  be  summarized  as  follows: 

•  Avionics  specialists  using  TOs  compared  with  those  using  IMIS. 

The  avionics  specialists  using  IMIS  found  more  correct  solutions  in  less  time, 
used  fewer  parts  to  do  so,  and  took  less  time  to  order  these  parts.  All  these 
results  were  statistically  significant. 

The  number  of  parts  required  deserves  brief  comment.  Savings  in  spare  parts 
inventory  and  transportation  were  by  far  the  largest  factors  in  the  Tomasetti  et 
al.  (1993)  cost  analysis.  The  number  of  parts  required  also  exerted  consider¬ 
able  leverage  on  the  overall  cost  savings  reported  by  Teitelbaum  and  Orlansky 
(1996). 

The  results  concerning  time  to  order  parts  are  to  be  expected  because  IMIS 
automates  most  of  this  process.  These  results  are  included  here  because  they 
are  large  and  because  the  time  required  by  technicians  to  complete  the  paper¬ 
work  in  the  absence  of  IMIS  could  be  spent  elsewhere  (i.e.,  substantial  pro¬ 
ductivity  gains  and  cost  savings  could  be  realized  if  IMIS,  or  a  similar 
capability,  performs  these  paperwork  chores). 
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Table  3.  Maintenance  Performance  of  12  Air  Force  Avionics  Specialists  and 
12  General  (APG)  Technicians  Using  Task  Orders  (TOs)  and  IMIS 


Correct 

Solutions 

(%) 

Time  To 

Solution 

(Min) 

Average 
Number  of 

Parts  Used 

Time  To 

Order  Parts 
(Min) 

TOs 

IMIS 

TOs 

IMIS 

TOs 

IMIS 

TOs 

IMIS 

Avionics  Specialists 

81.9 

100.0 

149.3 

123.6 

8.7 

19.4 

1 .2 

APG  Technicians 

69.4 

98.6 

175.8 

124.0 

8.3 

5.3 

25.3 

1.5 

•  APG  technicians  using  TOs  compared  with  those  using  IMIS. 

Thomas  found  similar  results  in  these  comparisons.  The  APG  technicians 
using  IMIS  found  more  correct  solutions  in  less  time,  used  fewer  parts  to  do 
so,  and  took  less  time  to  order  them.  As  with  the  avionics  specialists,  all  these 
results  were  statistically  significant. 

•  APG  technicians  using  IMIS  compared  with  avionics  specialists 
using  TOs.  The  APG  technicians  using  IMIS  found  more  correct  solutions 
in  less  time,  used  fewer  parts  to  do  so,  and  took  less  time  to  order  these  parts 
than  the  avionics  specialists  using  paper-based  TOs.  All  these  results  were  sta¬ 
tistically  significant.  This  result  suggests  that  replacing  some  of  the  extra 
training  required  by  specialists  with  the  on-the-job,  just-in-time  performance 
aids  (e.g.,  IMIS)  supplied  to  nonspecialists  is  feasible  and  desirable. 

•  APG  technicians  using  IMIS  compared  with  avionics  specialists 
using  IMIS.  In  these  comparisons,  the  APG  technicians  performed  almost 
as  well  as  the  Avionics  specialists  overall  and  even  slightly  better  in  the  num¬ 
ber  of  parts  used.  None  of  these  comparisons  were  statistically  significant,  and 
none  appear  to  be  practically  significant.  These  results  again  suggest  the  feasi¬ 
bility  of  replacing  some  number  of  specialists  (and  their  greater  training  costs 
and  requirements)  with  general  technicians  who  are  provided  with  on-the-job, 
just-in-time  performance  aids.  These  results  also  suggest  the  desirability  of 
doing  so,  because,  in  this  case,  the  training  costs  of  the  specialists  are  greater 
than  those  of  the  nonspecialists  even  though  the  resulting  performance  on  the 
job,  where  it  counts,  is  the  same  in  both  cases. 

The  promise  suggested  by  these  results  could  vanish  if  the  costs  to  provide  the 
IMIS  performance  aid  exceed  the  costs  they  otherwise  save.  However,  what  if  the  costs 
and  benefits  analysis  by  Tomasetti  et  al.  (1993)  were  combined  with  the  empirical  results 
reported  by  Thomas  (1995)?  By  using  these  two  sources  of  data,  Teitelbaum  and  Orlansky 
(1996)  were  able  to  estimate  reductions  in  depot-level  maintenance,  organizational-level 
maintenance,  and  maintenance  and  transportation  of  inventories  of  spare  parts.  They 
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estimated  annual  savings  from  the  use  of  IMIS  to  be  about  $38  million  for  the  full  Air 
Force  fleet  of  about  1,700  F-16s. 

Teitelbaum  and  Orlansky  also  considered  the  costs  to  develop  and  maintain  IMIS. 
Assuming  an  8-year  useful  life  for  IMIS,  they  arrived  at  a  figure  of  about  $18  million  per 
year  to  maintain  IMIS  (including  its  databases)  and  to  amortize  its  development  costs.  The 
result  is  a  benefit  of  about  $20  million  per  year  in  net  savings. 

This  figure  of  $20  million  is  conservative.  It  does  not  include 

•  Savings  in  selection  and  training  that  would  result  from  a  reduction  in  Air 
Force  requirements  to  recruit  and  train  specialized  personnel  (e.g.,  the  avionics 
specialists  in  Thomas’  study) 

•  Savings  in  training  that  would  accrue  from  using  IMIS  as  a  performance  aid 
and  as  a  training  device 

•  Savings  in  the  costs  to  print,  distribute,  and  update  paper  TMs 

•  Savings  (of  about  50  percent)  in  time  to  debrief  pilots  about  maintenance 
problems. 

Most  importantly,  these  benefits  do  not  include  those  arising  from  increased  sortie  rates  and 
unit  operational  readiness  and  effectiveness  that  would  result  from  the  substantially 
improved  problem-solving  competencies  of  maintenance  personnel.  The  monetary  value  of 
these  benefits  is  difficult,  perhaps  even  impossible,  to  assess. 

C.  DISCUSSION  AND  CONCLUSIONS 

At  least  six  observations  can  be  made  concerning  the  findings  reported  in  this 
document. 

1 .  Observation  1:  Performance  aids  enhance  performance  and  lower  costs. 

The  Oncology  Aid,  CMAS,  PEAM,  and  IMIS  assist  decision-making  and  problem 
solving.  Capabilities  such  as  job  performance  aids,  electronic  performance  support  sys¬ 
tems,  technology-based  “mentoring,”  and  those  capabilities  more  typically  described  as 
individual  and  group  performance  aids  are  intended  to  match  user  intentions  and  relevant 
data  with  the  decision  heuristics  that  address  a  full  range  of  problem  solving.  As  suggested 
by  results  presented  in  this  document,  this  range  includes  the  maintenance  of  devices  and 
systems.  Performance  aiding  in  maintenance  operations  has  yielded  useful  evidence  on 
effectiveness  and  cost  returns.  This  evidence  indicates  the  general  value  of  technology- 
based  performance  aiding. 
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2 .  Observation  2:  Design  guidelines  are  needed. 

The  results  discussed  in  this  document  suggest  that  a  strong  cost-effectiveness  case 
can  be  made  for  the  development  and  implementation  of  technology-based  performance 
aids  across  a  variety  of  applications.  More  data  of  this  sort  are  needed  for  a  conclusive 
case,  but,  as  current  findings  suggest,  so  far,  so  good.  What  is  not  clearly  evident  is  how 
these  technology-based  capabilities  should  be  designed.  Current  functional  designs  are 
based  on  best  guesses.  We  still  have  much  to  learn  about  what  functionalities  should  be 
included  to  ensure  that  these  technology-based  capabilities  serve  as  effective  partners  in 
human  decision-making  and  problem  solving.  To  accomplish  this  end,  we  need  to  know 
more  about  both  the  functionalities  we  are  able  create  and  the  human  problem-solving  proc¬ 
esses  we  want  to  assist. 

3  .  Observation  3:  Performance  aids  are  part  of  a  system. 

A  decision,  for  instance,  to  supply  IMIS  to  all  Air  Force  APG  technicians  may  be  a 
good  idea,  but  it  should  not  be  extended  to  a  wholesale  replacement  of  all  avionics  special¬ 
ists  and  avionics  specialist  training.  This  decision  should  consider  the  full  perspective  of  all 
efforts  undertaken  to  ensure  that  human  performance  is  provided  when  and  where  it  is 
needed. 

More  generally,  aids  for  performance  aiding  and  problem  solving  need  to  be  treated 
as  components  of  an  integrated  system  intended  to  ensure  the  availability  of  human  compe¬ 
tence.  The  object  is  not  just  effective  decisions  or  problem  solving  alone,  but  also  an  effec¬ 
tive  military  organization.  Resources  to  accomplish  this  end  involve  developing  selection 
standards  for  the  people  who  are  to  solve  the  problems;  structuring  the  tasks,  jobs,  and 
careers  to  which  they  are  assigned;  providing  training  and  education;  and,  of  course,  care¬ 
fully  designing  the  performance  aids  they  will  use  and  ensuring  the  effective  implemen¬ 
tation  of  these  performance  aids.  All  these  components  interact.  An  investment  (or  lack 
thereof)  in  one  area  affects  all  other  components  and  the  functioning  of  the  organization  as  a 
whole.  Determining  these  allocations  should  be  treated  as  part  of  the  full  system  of  human 
competence  needed  by  the  organizational  entity— be  it  a  military  unit,  company,  university, 
or  government  agency. 
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4 .  Observation  4:  Performance  aids  should  accommodate  natural  and  for¬ 
mal  heuristics. 

As  Edwards  and  Fasolo  (2001)  discuss,  the  design  of  performance  aids  has  neces¬ 
sary  roles  for  descriptive  and  prescriptive  approaches.  Prescriptive  theories  help  by 
explaining— often  in  quite  formal  terms  that  are  amenable  to  algorithmic  procedures— how 
decisions  should  be  made,  based  on  well-defined  criteria  and  optimized  choice  of  alterna¬ 
tives.  Techniques  like  utility  theory  (von  Neumann  and  Morgenstem,  1947)  rational  choice 
theory  (Simon,  1955),  and  prospect  theory  (Kahneman  and  Tversky,  1979)  are  all  applica¬ 
ble.  Edwards  and  Fasolo  summarized  current  prescriptive  techniques  by  organizing  them 
under  three  widely  used,  general  approaches  (multiattribute  utility  measurement,  Bayesian 
probability  rules,  and  maximization  of  subjectively  expected  utility),  all  of  which  can  play  a 
role  in  good  decision-making. 

In  contrast,  descriptive  approaches  attempt  to  explain  how  people  actually  make 
decisions  in  “real  life.”  These  approaches  often  use  case  studies  to  understand  and  explain 
decisions  by  describing  the  actors,  the  context  of  the  decisions,  and  the  intended  outcomes 
for  case.  Examples  of  this  second  category  include  an  analysis  of  decision-making  in  the 
Cuban  missile  crises  (Allison,  1971)  and  an  analysis  of  the  surprise  attack  at  Pearl  Harbor 
(Wohlstetter,  1962).  Such  descriptive  approaches  were  used  extensively  in  the  design  of 
the  Oncology  Aid. 

A  synthesis  of  descriptive  and  prescriptive  approaches  can  be  provided  by  natural¬ 
istic  decision-making  (NDM)  (e.g.,  Orasanu  and  Connolly,  1993;  Klein,  2000).  NDM 
combines  elements  of  formal  models  with  reason-based  analysis,  elicitation,  and  direct 
observation.  Zsambok  (1997)  describes  NDM  as  the  “way  people  use  their  expertise  to 
make  decisions  in  field  settings”  (p.  4).  It  attempts  to  capture  and  describe  decision-making 
by  observing  and  interviewing  individuals  (as  in  descriptive  approaches)  and  abstracting 
from  these  cases  more  formal  models  (as  in  prescriptive  approaches),  such  as  Klein’s  Rec¬ 
ognition-Primed  Decision  Model  (Klein,  Calderwood,  and  Clinton-Cirocco,  1985),  which 
categorizes  decision-making  stages  and  strategies.  A  systematic  effort  to  apply  NDM  to  the 
maintenance-aiding  applications  discussed  in  this  document  may  be  an  important  next  step 
in  the  development  of  these  applications. 

Performance  aids  need  to  address  real-world  decision  processes  and  the  require¬ 
ments  of  field  applications  while  incorporating  prescriptive  theories  and/or  optimal  heuris¬ 
tics  derived  from  value-based  models.  Both  descriptive  and  prescriptive  approaches  can 
and  should  be  used  in  designing  and  developing  performance  aids.  As  the  examples 
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discussed  in  this  document  suggest,  computer  technology  may  not  be  necessary  to  support 
such  applications,  but  it  is  difficult  to  imagine  any  practical  alternative  to  it. 

5 .  Observation  5:  Performance  aids  need  “intelligence.” 

If  we  seek  technologies  that  participate  as  partners  in  human  problem  solving,  the 
technologies  need  to  understand  the  human  side  of  the  issue.  To  some  extent,  they  need  to 
be  “intelligent.”  For  instance,  the  primary  need  for  maintenance  technicians  (and  other 
problem  solvers)  is  not  a  capability  that  starts  at  the  beginning  of  a  procedure  and  leads 
them  through  to  the  end,  valuable  as  that  may  be.  More  typically,  maintenance  technicians 
begin  troubleshooting  a  specific  procedure,  encounter  anomalies,  and  need  help.  They  need 
a  device  that  has  the  capability  to  engage  in  a  decision-aiding,  mixed-initiative  dialogue  with 
either  the  technician  or  the  performance  aid  capable  of  taking  the  initiative  to  ask  questions, 
seek  clarification,  access  databases,  and  suggest  measurements  and  hypotheses.  Much  is 
made  of  dialogue  management  in  tutorial  instruction  (e.g.,  Graesser,  Person,  and 
Magliano,  1995).  A  capability  for  dialogue  management  in  problem  solving,  in  general, 
and  maintenance  aiding,  specifically,  is  needed  and  should  be  developed.  This  need  sug¬ 
gests  that  the  decision-aiding  communities  and  the  intelligent-tutoring  communities  should 
collaborate. 

Other  “intelligence”  is  required  in  technology-based  problem  solving  and  perform¬ 
ance  aiding.  As  described  by  Fletcher  (2002),  among  many  others,  this  intelligence  is 
needed  for  comprehensive  coverage  of  the  decision  space  so  that  the  suggested  actions  are 
relevant  and  applicable.  Intelligence  is  also  needed  to  represent  the  user  so  that  advice  is 
given  in  a  form  that  the  user— at  whatever  level  of  knowledge,  intent,  or  ability— is  capable 
of  understanding  and  using.  Finally,  heuristics  are  needed  to  infer  solutions  to  the  problem 
presented.  These  capabilities  are  to  one  degree  or  another  present  in  intelligent  tutoring  sys¬ 
tems  and  performance  aids.  It  is  not  a  great  distance  from  the  avionics  training  capabilities 
impressively  demonstrated  by  an  intelligent  tutoring  system  such  as  Sherlock  (Gott,  Kane, 
and  Lesgold,  1995)  to  avionics  performance  aiding.  Again,  the  communities  concerned 
with  these  developments  would  benefit  from  increased  coordination  and  communication. 

6 .  Observation  6:  Where  are  the  performance  aids? 

In  contrast  to  the  favorable  findings  reported  in  this  document  and  the  promise  of 
even  more  capabilities  as  technology-based  performance  aids  are  developed,  only  one  of 
the  devices— the  Oncology  Decision  Aid— is  currently  in  use.  CMAS,  PEAM,  IMIS,  and 
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Sherlock  are  absent  from  daily  maintenance  practice  in  the  military.  The  research  and 
development  (R&D)  community  has  assumed  a  responsibility  to  advance  the  state  of  the  art 
and  has  been  successful  in  fulfilling  it.  The  complementary  responsibility  to  advance  the 
state  of  practice  in  performance  aiding  and  problem  solving  through  technology  transfer 
and  engineering  in  the  field  does  not  appear  to  be  receiving  the  attention  it  needs. 

D.  FINAL  WORD 

This  brief  summary  presages  an  evolving  and  perhaps  inevitable  future  in  which 
hand-held,  or  more  likely  wearable,  personal  technology-based  learning  and  problem¬ 
solving  assistants  will  be  as  common  as  wristwatches.  They  will  be  widely  used  to  aug¬ 
ment  human  cognition  and  enhance  human  competency.  As  evidenced  by  the  findings 
reported  in  this  document,  we  can  build  technology-based  tools  that  make  us  “smart” 
(Norman,  1993).  We  will  communicate  with  them  in  natural  language,  and  they,  in  turn, 
will  communicate  with  the  global  grid  to  provide  advice  and  information.  How  well  they 
articulate  this  advice  and  information  back  to  the  individuals  who  use  them  will  depend,  to 
some  extent,  on  how  well  they  understand  each  individual’ s  needs,  intentions,  and  capa¬ 
bilities.  As  suggested  in  this  document  and  elsewhere,  capabilities  for  individualizing  pres¬ 
entations  and  communications  are  evolving  in  several  domains  [e.g.,  human  computer 
interaction,  modeling  and  simulation  (M&S),  and  intelligent  tutoring]. 

How  well  these  technology-based  learning  and  problem-solving  assistants  enhance 
human  problem  solving  will  also  depend  on  how  well  those  who  design  and  build  them 
understand  the  processes  human  beings  use  to  solve  problems.  What  is  the  optimal,  most 
effective  division  of  labor  between  humans  and  machines?  Meehl’s  (1954)  ancient  finding 
still  stands:  procedures  that  capture  our  decision  processes,  but  avoid  human  distractions 
and  foibles,  may  make  better  decisions  than  we  do.  We  must  learn  how  best  to  develop  and 
use  this  capability.  We  need  to  progress  from  art  to  engineering.  The  results  presented  here 
suggest  that  efforts  to  do  so  will  be  worthwhile. 
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